Research to Practice in School Psychology: Challenges Ahead and the Role of NASP’s School Psychology Forum

The goal of School Psychology Forum is to promote and disseminate research-topractice scholarship for the benefit of school psychologists in their clinical practice. This goal has evolved from a desired practice to a mandatory component of any clinical practice. Research to practice is of importance as the concept of evidence-based practice is codified in federal law, American Psychological Association standards for practice, and the Canadian Psychological Association standards for practice. Although there is a consensus that evidence-based practice presents a positive direction for service delivery in school psychology, there remains a stubborn and significant gap between research and clinical practice. Reasons for this gap are identified. In addition, suggestions for improving the credibility, robustness, and approaches to evaluation and implementation of clinical research are provided. School Psychology Forum is an excellent outlet for the publication of credible studies with strong research design to narrow the gap between research and practice. When School Psychology Forum was proposed by the National Association of School Psychology (NASP) president Bill Pfohl, the concept was that school psychologists would benefit from highquality research articles that were directly applicable to the practice of school psychology. Evidencebased practice is now part of multiple laws in education (West, n.d.) and this requires access to highquality research. School Psychology Forum is on the way to becoming a leader in realizing Dr. Pfohl’s vision of a NASP resource that supports school psychologists’ implementation of cutting-edge research for the benefit of school children, families, schools, teachers, and communities. Scholarly journals have traditionally focused on promoting research that supports and tests theory, otherwise known as basic or blue sky research (Ioannidis, 2016a). To repurpose scholarly journals to support evidence-based practices in psychological and educational settings is a structural challenge. Based on my experience as the editor of School Psychology Forum since 2010, I have a set of stages to be applied before scholarly journals can be a relevant source of information for the implementation of evidencebased practices. A major barrier is that most psychologists do not read professional journals (American Psychological Association Presidential Task Force on Evidence-Based Practice, 2006). This is a reasonable response to scholarly journals being indifferent or hostile to research relevant to clinical practice. Most academics and professional researchers are evaluated and receive tenure on the basis of their publication in high-impact journals. Nearly all of the highest impact journals in psychology and education focus exclusively on theory and theory-based research. Clinical and implementation research is most often published in journals with lower impact. Thus, most researchers, including Correspondence concerning this article should be directed to Steven R. Shaw, McGill University, 3700 McTavish, Montreal, QC H3A 1Y2 Canada; [email protected]. Copyright 2016 by the National Association of School Psychologists, ISSN 1938-2243 those working directly in school psychology preparation programs, have an incentive to publish articles that are not directly related to school psychology practice. In addition, there is an incentive for currently low impact factor journals aspiring to status as high impact journals to publish papers that are primarily tests of theory. Without scholarly journals publishing clinically relevant articles, there is no incentive for school psychologists to read professional scholarly journals. The situation sets up a common, but false, dichotomy of research versus clinical or theoretical versus practical information. Practitioners want to have access to clinical and practical information and often are not interested in research or theoretical information. Most published research has a high degree of internal validity through strong research design, but lacks relevance and consideration of clinical applications (i.e., external validity). As such, research lacks credibility for clinicians (Gambrill, 2006). Credibility may be as important of an issue as internal and external validity when it comes to scientific support for evidence-based practices (Ioannidis, 2005). REFINING RESEARCH TO PRACTICE Research to practice is an extraordinarily complex enterprise. The mechanisms of the scientistpractitioner model that effectively support research to practice are not entirely clear (Joyner, Paneth, & Ioannidis, 2016). There are many factors and issues to be addressed by both researchers and clinicians before research to practice is a seamless and productive tradition in school psychology. Evidence-based practice is a process involving the “conscientious, explicit, and judicious use of the best available research evidence to inform each stage of clinical decision making and service delivery” (Canadian Psychological Association Task Force, 2012, p. 7). This approach to clinical practice in medicine was introduced in a seminal article (Evidence-Based Medicine Working Group, 1992), which represented the formal introduction of a new decision-making paradigm requiring clinicians to learn new skills based on scientific research in order to provide better care to patients or clients. Codified in U.S. law, accreditation of training programs and internship sites, and part of the new tradition of the practice of psychology, evidence-based practices are now the standard for all service delivery in school psychology (West, n.d.). The exact definition and operationalization of what is meant by evidencebased practice is not always clear. There are opportunities for misinterpretation or deviation from the original intent of evidence-based practice. Without a strong operational definition, evidence-based practice has become a halo term meaning all things good or a seal of approval for all interventions and practices (Thurlings, Evers, & Vermeulen, 2015). This has been referred to as a hijacking of the concept (Ioannidis, 2016b). For-profit consultation, manualized interventions, and sales of products such as tests or curricula rely on seeking the status of being labeled as an evidence-based practice for their sales and business. Frequently, studies are cherry picked to report only studies showing a statistically significant support. The use of evidence-based practices by these products frequently ignore issues such as the quality of study design and whether the studies are independent of those with a financial interest in the outcome (Forman, Olin, Hoagwood, Crowe, & Saka, 2009; Ioannidis, 2016b). The use of the phrase evidence-based practice as a marketing tool is clearly not the intent of the concept. However, given the difficulties and complexities of applying evidence-based practice, aggressive marketing campaigns using the phrase can overwhelm consumers and the result is the hijacking of an important idea (Shaw, Boulanger, & Gomes, 2015). At the other end of the spectrum there is the hard interpretation of evidence-based practice, which is that every intervention and professional practice must have overwhelming or at least consensus support from experimental research studies. In some cases, the hard interpretation also involves implementing interventions using the same methodology as was used in the published research study (Karthikeyan & Pais, 2010). This hard interpretation has led to charges that evidence-based practice is a robotic NASP | School Psychology Forum: Research in Practice Research to Practice | 341 approach to clinical practice that robs psychologists of professionalism. The purpose of the concept is to have research inform clinical practice, not to place a straightjacket on practitioners and prevent them from meeting the needs of their clients (La Greca, Silverman, & Lochman, 2009). Both the hijacked and the hard interpretation of evidence-based practice do not accurately describe the original intent and definition of the construct. Yet, exactly how evidence-based practice is applied in real clinical activities remains to be clarified. There are many opportunities and challenges to the full and correct implementation of evidence-based practices. Among the most important areas are improvement in clinical research and research-to-practice efforts. School Psychology Forum has potential to be an important outlet for improved information that is directly relevant to practitioners. However, there is a host of specific issues to be addressed before the groundwork for widespread evidence-based practice implementation can take place. REPRODUCIBILITY/REPLICATION CRISIS One of the difficulties with evidence-based practice is the requirement that there be a scientific basis for the effectiveness of the practice on outcomes. A practice cannot have a scientific basis unless it meets the minimum requirements of being reproduced and replicated (Greenland et al., 2016). In a large study attempting to replicate classic and important studies in psychology, most replications failed to find effect sizes as large as the original published study (Open Science Collaboration, 2015). However, most intervention studies do not even meet the minimum standards of demonstrating that they can be reproduced (Stephens, n.d.). Reproduction involves the same researchers under the same general conditions evaluating a practice exactly as in the original study and finding the same results. This is a minimum standard for scientific findings (Greenland et al., 2016). Replication involves independent investigators, methods, equipment, and protocols attempting to determine whether an effect is robust and can be found under slightly different conditions. In order for a clinical research study to be relevant and useful, there must be evidence that the effect can be reproduced and that it can be replicated (Shaw & D’Intino, in press). Without this exercise there is no scientific support for what we call evidence-based practice. Very few published research findings meet the criterion of reproducibility and even fewer meet the criterion of replicability (Coyne, Cook, & Therrien, 2016). With replicability being a necessary condition for being considered an evidence-based practice, the construct of the robustness of a finding is essential (Baker, 2016). That is, how much can the context, sample, methods, assessment, and other variables (i.e., treatment integrity) vary from the original study and still show the same positive effect size. An original and novel study that shows a large and positive effect size is nothing more than a proof of concept. The degree to which this large and positive effect size can survive various types of replication indicates the usefulness of the intervention for clinical application and the robustness of the effect size (Levant & Hasan, 2008). EVALUATING CLINICAL RESEARCH Evaluating clinical research is a critical aspect of all evidence-based practice. There are often deep disagreements on the quality and utility of published research among professional scholars who conduct and evaluate research full time (Rycroft-Malone et al., 2004). Expecting full-time clinicians to make these evaluations and judgments is not only a burden, but unrealistic and an inefficient use of clinician time. There are some resources that help with the evaluation of the quality of research. For example, Cochrane Reviews are systematic evaluations of published research in medicine and healthcare policy. Cochrane Reviews describes themselves as “internationally recognized as the highest standard in evidence-based healthcare resources. They investigate the effects of interventions for prevention, treatment, and rehabilitation” (Cochrane, n.d.). In the field of education, NASP | School Psychology Forum: Research in Practice Research to Practice | 342 [t]he What Works Clearinghouse reviews the existing research on different programs, products, practices, and policies in education. Our goal is to provide educators with the information they need to make evidence-based decisions. We focus on the results from high-quality research to answer the question “What works in education?” (italics in the original text). (What Works Clearinghouse, n.d.) These tools provide basic information for clinicians to evaluate published and some gray literature. Evaluating the quality of research design and effect sizes is not sufficient in the evaluation of clinical research (Anjum, 2016; Evidence-Based Medicine Working Group, 1992). Much like the construct of validity, utility of research depends greatly on context. This is similar to the reproducibility and replicability problem. For clinicians, the value of the study may involve determining whether the sample used in the published study is similar to the target audience of the intervention in question. Sampling match is a major issue. Other issues concern resources required for implementation, cultural appropriateness of the intervention, feasibility, student centeredness, acceptability by teachers and parents, and context (Biesta, 2007). Even the highest quality studies that achieve the highest marks on Cochrane Reviews or What Works Clearinghouse lack relevance and credibility for practice unless all of these variables are considered (May, Johnson, & Finch, 2016). Another issue concerns publication bias. There is clearly bias in published research due, in part, to the desire among journal editors to publish novel and exciting ideas with large effect sizes (Lee, Sugimoto, Zhang, & Cronin, 2013). As such there is little motivation to publish replications or studies that show no statistically significant findings (Franco, Malhotra, & Simonovits, 2014). Often, published findings have extremely large sample sizes, which give them statistical power to identify small effects. The result is that there are many studies published with findings significant at the p , .05 level, but with extremely small effect sizes, which indicate minimal practical effects on outcomes (Cohen, Manion, & Morrison, 2013; Makel & Plucker, 2014). Studies often have statistically significant, but not practically important, findings. In addition, clinical research studies showing no significant effects are typically not published in refereed professional journals (Ferguson & Heene, 2012). In the hierarchy of evaluation of evidence-based practices, meta-analyses and comprehensive literature reviews are considered to be the highest state of the literature (Evidence-Based Medicine Working Group, 1992). The logic is that multiple studies that all point in the same direction create a more compelling case supporting an intervention than does a single study. However, given the publication bias that is common in educational and psychological research literature, meta-analyses and comprehensive literature reviews may be simply an accumulation of biased studies and may not necessarily provide compelling evidence that an intervention or practice has an overall positive effect on outcomes (Dwan, Gamble, Williamson, & Kirkham, 2013). IMPROVING CLINICAL RESEARCH An essay by Ioannidis (2016a) considers clinical research, such as randomized controlled trials of interventions, and concludes that most of them are no more useful for clinical practice than basic science or blue-sky research. Based on this evaluation of clinical research, journals purporting to publish research-to-practice articles, such as Journal of Applied School Psychology and School Psychology Forum, may not provide refereed research studies containing information with any clinical utility. In the discussion of medical research, several features required for clinical research to be considered useful and credible are described. Problem base (Is the problem being addressed a major issue with significant consequences for large numbers of people?), information gain (How much information does the new study contribute in addition to what is already known?), pragmatism (Does the research design reflect real life?), patient centeredness (Does the clinical research reflect priorities of stakeholders?), value for the money (Does the intervention require significant financial resources in time, training, materials, and NASP | School Psychology Forum: Research in Practice Research to Practice | 343 time lost that could be spent on other activities that make implementation of the intervention prohibitive?), feasibility (Can the intervention described in the research be carried out?), and transparency (Are the methods, data, and analyses verifiable and unbiased?), all these features described are for evidence-based practice in medicine. These features also apply to education and psychology, yet there are also additional features required for clinical research to be useful and credible. In addition to the above features, for clinical research to be credible in school psychology evidencebased practice there must also be the features described in Table 1. Clinical research will continue to be less than useful and not credible unless a model of research is developed that addresses these fundamental features. IMPLEMENTATION SCIENCE Knowing what works and under what circumstances an intervention is effective is only part of establishing evidence-based interventions. Knowing exactly how to implement an intervention is just as important as knowing what interventions are effective (Durlak & DuPre, 2008). Transforming interventions with research support into an idea that is implemented in a counseling group or classroom is difficult. Implementation science and related research investigate and address major bottlenecks that impair effective implementation of interventions, test new methods to improve academic and mental health adoption, and help to determine the relationship between interventions and outcomes (Klein & Knight, 2005). Currently, implementation research in school psychology has focused primarily on the construct of treatment integrity; that is, the degree to which methods of implementation match the methods agreed upon and designed by student support teams or other professionals (Moullin, Sabater-Hernández, Fernandez-Llimos, & Benrimoj, 2015). In the case of evidence-based practices, the methods agreed upon and designed are based on published research with evidence of positive effect sizes. Although treatment integrity is an extremely valuable tool in implementation, it does not embrace all of the options and factors that are considered in implementation science. Implementation science is a new label, but in the field of school psychology this category of investigation has been integrated into professional practice and consultation literature. Integrating extant literature with new implementation processes that are developed primarily in the field of medicine have potential Table 1. Additional Features for Credibility of Educational Research Legally consistent Characteristics of the intervention described in the research consistent with federal, state, and local legislation, case law, and